Gate output-centric image_to_nchw staging to discrete GPUs (#21022)#21022
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21022
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New Failure, 1 Unrelated FailureAs of commit 5605479 with merge base 21554e5 ( NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@SS-JIA has exported this pull request. If you are a Meta employee, you can view the originating Diff in D112599710. |
This PR needs a
|
37460f1 to
0fbb318
Compare
…1022) Summary: The `image_to_nchw` output-staging shader can dispatch one thread per NCHW output element so writes to the host-visible staging buffer are fully coalesced. On a discrete GPU the staging buffer is host-visible but not device-local (PCIe-backed system RAM), where coalescing writes is a large win: ~11x end-to-end on an RTX 4080 laptop for an ESPCN 2x super-resolution model. That output-centric scheme fetches each texel up to 4 times (once per component). On unified-memory (mobile) GPUs the staging buffer is not PCIe-backed, so write coalescing buys nothing and the redundant fetches are a net loss: measured +15-42% `image_to_nchw` per-dispatch on Mali-G715 and +3-6% on Adreno 750. Gate the two strategies on `Adapter::has_unified_memory()`: - discrete (no unified memory) -> `image_to_nchw_coalesced_*` (output-centric) - unified memory (mobile) -> `image_to_nchw_*` texel-centric (one thread per texel, single fetch, contiguous NCHW writes), which is the default. `image_to_nchw.glsl` emits both variants via a `COALESCED_WRITES` codegen flag (default False). `get_tensor_to_nchw_shader` selects the coalesced variant when `!has_unified_memory()`; the global-workgroup-size picker keys off the shader name so the dispatch topology matches the chosen variant. Reviewed By: metascroy Differential Revision: D112599710
…1022) Summary: The `image_to_nchw` output-staging shader can dispatch one thread per NCHW output element so writes to the host-visible staging buffer are fully coalesced. On a discrete GPU the staging buffer is read back over PCIe, where coalescing writes is a large win: ~11x end-to-end on an RTX 4080 laptop for an ESPCN 2x super-resolution model. That output-centric scheme fetches each texel up to 4 times (once per component). On integrated (mobile / on-die) GPUs the staging buffer is not PCIe-backed, so write coalescing buys nothing and the redundant fetches are a net loss: measured +15-42% `image_to_nchw` per-dispatch on Mali-G715 and +3-6% on Adreno 750. Gate the two strategies on the physical device type via `Adapter::is_integrated_gpu()`: - discrete (not integrated) -> `image_to_nchw_coalesced_*` (output-centric) - integrated (mobile) -> `image_to_nchw_*` texel-centric (one thread per texel, single fetch, contiguous NCHW writes), which is the default. Gate on device type, NOT `has_unified_memory()`: a discrete GPU with Resizable BAR exposes a DEVICE_LOCAL | HOST_VISIBLE memory type, so `has_unified_memory()` returns true for it and would wrongly route it to the slow texel-centric path (measured ~10x end-to-end regression on an RTX 4080 laptop). `image_to_nchw.glsl` emits both variants via a `COALESCED_WRITES` codegen flag (default False). `get_tensor_to_nchw_shader` selects the coalesced variant when `!is_integrated_gpu()`; the global-workgroup-size picker keys off the shader name so the dispatch topology matches the chosen variant. Reviewed By: metascroy Differential Revision: D112599710
0fbb318 to
5605479
Compare
Summary:
The
image_to_nchwoutput-staging shader can dispatch one thread per NCHWoutput element so writes to the host-visible staging buffer are fully coalesced.
On a discrete GPU the staging buffer is read back over PCIe, where coalescing
writes is a large win: ~11x end-to-end on an RTX 4080 laptop for an ESPCN 2x
super-resolution model.
That output-centric scheme fetches each texel up to 4 times (once per
component). On integrated (mobile / on-die) GPUs the staging buffer is not
PCIe-backed, so write coalescing buys nothing and the redundant fetches are a
net loss: measured +15-42%
image_to_nchwper-dispatch on Mali-G715 and +3-6%on Adreno 750.
Gate the two strategies on the physical device type via
Adapter::is_integrated_gpu():image_to_nchw_coalesced_*(output-centric)image_to_nchw_*texel-centric (one thread per texel,single fetch, contiguous NCHW writes), which is the default.
Gate on device type, NOT
has_unified_memory(): a discrete GPU with ResizableBAR exposes a DEVICE_LOCAL | HOST_VISIBLE memory type, so
has_unified_memory()returns true for it and would wrongly route it to the slow texel-centric path
(measured ~10x end-to-end regression on an RTX 4080 laptop).
image_to_nchw.glslemits both variants via aCOALESCED_WRITEScodegen flag(default False).
get_tensor_to_nchw_shaderselects the coalesced variant when!is_integrated_gpu(); the global-workgroup-size picker keys off the shadername so the dispatch topology matches the chosen variant.
Reviewed By: metascroy
Differential Revision: D112599710